Skip to content

Harden shared CI against transient failures - #437

Open
sjmiller609 wants to merge 18 commits into
mainfrom
sjmiller609/test-flakes
Open

Harden shared CI against transient failures#437
sjmiller609 wants to merge 18 commits into
mainfrom
sjmiller609/test-flakes

Conversation

@sjmiller609

@sjmiller609 sjmiller609 commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

summary

  • retry checkout and SDK lint validation after transient network or dependency-download failures
  • derive test bridge names from uniquely leased subnets to prevent cross-process bridge collisions without reducing parallelism
  • recover and restage the host-provisioned Windows image and backing fixtures into /ci on every test run

tests

  • go test ./lib/instances -run ^TestBridgeNameForTestSubnet$ -count=1
  • git diff --check

Note

Medium Risk
Touches shared self-hosted CI (checkout, fixture copies as root, lint retries) and test Linux networking. Failures could skip Windows tests or leave flaky CI rather than production auth/data paths.

Overview
Hardens shared CI against transient checkout/lint failures, missing Windows VM fixtures, and colliding test network bridges.

Checkout now retries once on failure. SDK lint in STLC generation retries up to three times. Test jobs copy host-provisioned Windows images into /mnt/data/ci-fixtures/windows and symlink them under /ci/windows.

Test Linux bridges are named from the leased subnet (hm + octets) instead of a seed/seq combo, so parallel processes no longer share a bridge name.

Reviewed by Cursor Bugbot for commit bc23f12. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown
-->

✱ stlc build

go code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

116 files generated at d3f05cd (pushed)

go get github.com/kernel/hypeman-go-staging@d3f05cd6900a15e7d2e5d0e7527ef7fbdfab3d5a
python code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

230 files generated at 681b411 (pushed)

typescript code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

138 files generated at 7aa6c0d (pushed)

Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note
LevelCodeMessageTargets
Build metadata
Buildbd_76BZymHX-fearless-prong
Timestamp2026-08-21T18:49:52.617Z
stlc8413509
Spec hash0f15511e31a6
Config hash55e15f6f4434

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-08-21 18:50:22 UTC

@sjmiller609
sjmiller609 marked this pull request as ready for review August 21, 2026 12:56
Comment on lines +461 to +472
func TestBridgeNameForTestSubnet(t *testing.T) {
t.Parallel()

first, err := testBridgeNameForSubnet("10.200.1.0/24")
if err != nil {
t.Fatal(err)
}
second, err := testBridgeNameForSubnet("10.200.2.0/24")
if err != nil {
t.Fatal(err)
}
if first != "hmc801" || second != "hmc802" || first == second {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks like it's just trying two options but that would only decrease but not resolve the issue

bridgeName, err = testBridgeNameForSubnet(subnet)
if err != nil {
return err
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orphan bridges can block subnets

Medium Severity

Bridge names are now a fixed function of the leased subnet, but subnet selection only skips leases and overlapping routes. An orphan hm* bridge with no lease and no route still maps to the same name on the next lease of that subnet, so createBridge can fail when the interface already exists without the expected address. The old seed-based names avoided reusing that interface name.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 7e3924e. Configure here.

sudo cp --reflink=auto --sparse=always "$existing" "$source"
sudo chmod 0444 "$source"
fi
fi

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recovery find can abort CI

Medium Severity

The Windows fixture staging path is best-effort: missing source only skips the symlink. The new sudo find runs in that same path under Actions’ default bash -e, so a missing /mnt/data/home or any find error fails the whole step and blocks later tests that do not need the fixture.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 44df562. Configure here.

Comment thread .github/workflows/test.yml
Comment thread .github/workflows/test.yml

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 5 total unresolved issues (including 4 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 1202fd1. Configure here.

Comment thread .github/workflows/test.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant